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ABSTRACT 

In June 1993 the "Wall Street Journal" carried a 
table of data prepared by the Heritage Foundation that listed the 
states in order of the average amount they expend on each public 
school student. The table also contained each staters rank on the 
average score on the Scholastic Aptitude Test (SAT) and the rank of 
each state in the average performance of a random sample of eighth 
graders on the mathematics portion of the National Assessment of 
Educational Progress (NAEP) . The obvious inference drawn from this 
table was that spending money on education does not improve student 
performance, since some states that spent a great deal did not have 
high performance indices. Any such broad inference is almost surely 
specious. Setting aside such considerations as the differing cost of 
living, one issue is examined: self-selection. Average performance on 
the SAT does not represent study proficiency in that state, since 
only a small percentage of students take the SAT in some states. The 
uncertainty within a state about who takes the SAT is too great an 
obstacle for statistical adjustment to overcome. NAEP scores, which 
are closer to being the sort of random sample that is required for 
meaningful comparisons, are a more reasonable measure, and when they 
are examined with expenditures, there is a small relationship that 
indicates that spending money does improve performance. For every 
thousand dollars spent a state's NAEP ranking improves by two places. 
The reproduced table and two figures illustrate the discussion. 
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Does spending money on education help? 
A reaction to the Heritage Foundation and the Wall Street Journal 



Howard Wainer 
Educational Testing Service 

On Tuesday, June 22nd 1993, the Wall Street Journal carried a table of data 
prepared by the Heritage Foundation. It lists all of the states in order of the average 
amount that they expend on each public school pupil. Secondly, the table contains each 
staters rank on the average score on the Scholastic Aptitude Test (SAT) obtained by 
combining equally the SAT- Verbal and SAT-Mathematical scores for those students in 
that state who opted to take that exam. Last, it shows the rank of each state in the average 
performance of a random sample of their 8th grade students on the mathematics portion 
of the National Assessment of Educational Progress (NAEP). 



Insert Table 1 About Here 



It was observed that some states (like New York, New Jersey and Connecticut)" 
that have a high average per pupil expenditure don't rank very highly on the performance 
indices, and that conversely other states (like North Dakota and Iowa) that spend much 
less do very well indeed. The obvious inference is that spending money on education 
does not improve students' performance. Indeed quite the opposite. Is this inference borne 
out by the data presented? 

Shown in Figure 1 is a plot of Per pupil Expenditure vs. State Ranks on average 
SAT scores. We see a strong relationship. It appears that a state's SAT ranking worsens 
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by almost 6 places for every thousand dollars that was spent on the average pupil. What 
inferences can be drawn from this observation? 



Insert Figure 1 About Here 



I believe that any broad inferences drawn from this observation are almost surely 
specious. An obvious concern is uncertainty about the direction of the causal arrow. For 
example would anyone, even from the Heritage Foundation, upon viewing a similar plot 
that showed the positive relationship between money spent on medical care and mortality 
rate conclude that the way to a longer life is to reduce medical spending? 

Setting aside the logical difficulties with this inference there are so many other 
problems with these data that trying to even describe them would lengthen this note far 
beyond its purpose. There is an extensive scientific literature surrounding it and interested 
readers can begin in a recent review paper (Wainer, 1989) and follow the topic from the 
work referenced therein. In this discussion I will ignore the obvious problems associated 
with: 

1 . Differences in the cost of living between states. Does a dollar buy the same 
amount of anything in Alaska as it does in North Carolina? 

2. Using a state as the unit of discussion. Is the average per pupil expenditure the 
same in Manhattan as it is in Utica? 

3. Differences associated with the costs of education associated with various 
demographic mixtures. Do we expect to get the same SAT return on every dollar 
in a population heavy with recent Asian or Hispanic immigrants as we would in 
one whose native language is English? 
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4. Comparing one measure that averages over all individuals in a state (per pupil 
expenditures) with another that averages over a distinctly nonrandom subsample 
(SAT scores). 

Although all of these unaddressed issues can (and do) make inferences from such a gross 
comparison completely specious, I will ignore them here and focus attention on just one 
issue: self-selection. 

Average performance on the SAT in a state does not represent the students' 
proficiency in that state. In some states only a very small percentage of students opt to 
take it, in others a large proportion do* In Iowa only 3% whereas in Connecticut 78%. If 
we chose the top 3% from the Connecticut SAT-takers their average scores would 
smother the currently top-ranked Hawkeyes. Are the 3% of lowan high school seniors 
who take the SAT the best that Iowa has to offer? Who knows? I wouldn't make book on 
it, but we have no evidence. What would Iowa's rank be if 78% of their students took the 
test? Almost surely lower, but how much so? It is clear that examining the raw ranks will 
not get us anywhere. There have been many attempts to adjust these ranks statistically but 
none were successful . 

The uncertainty associated with who within a state decides to take the SAT has, so 
far, proven to be too great an obstacle for statistical adjustment to overcome. The 
uncertainty associated with self-selection yields bounds that are too broad for any useful 
purpose. In one study Connecticut's adjusted SAT rank was 1 and in another it was 35. 
Although the portion of the population sampled by the SAT is an important one, it is too 
poorly defined and insufficiently representative to be of any use in the sorts of 
comparisons suggested by the Heritage Foundation. 

NAEP scores are different. They are closer to being the sort of random sample 
that is required for meaningful comparisons. Of course the rankings given were for 8th 

7 




Page - 3 



graders not seniors, and they were for just mathematics, but they are a reasonable sample. 
A comparison of the state ranks based on NAEP scores with those based on SAT scores 
reveals that the relationship between the two, though positive, is very weak. Knowing a 
state's ranking on one measure helps to dissipate only 10% of one's uncertainty of that 
state*s ranking on the other. 

I do not view this as a shortcoming of the NAEP scores as a measure of a state's 
performance. Rather it puts an explicit hmit on the value of mean SAT scores as an 
educational indicator. 

Interestingly, if we use the more reasonable NAEP rankings as the dependent 
variable in trying to assess the relationship between average Per Pupil Expenditures and 
student performance we find a small relationship in a direction opposite that suggested by 
the Heritage Foundation. That for every thousand dollars spent a state's NAEP ranking 
improves by two places. 



Insert Figure 2 About Here 



I have not attempted to adjust statistically the estimates of the relationship 
between money spent and student performance, although it is certain that such 
adjustments would increase the size of the observed effect. My goal here was merely to 
indicate that the NAEP rankings are a more suitable dependent variable for such 
comparisons, and that even in the grossest possible kind of comparison we have seen that 
spending money is related to higher scores. 
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Table 1 



Money Dpesn'tHelp 



Wt table, producad by tha HarHaoa Foundation, compam tbo dollars spoM par ttodt nt In mh 
s3ate with trtat tUta*s ranking aceerdino to 199rs Sehottstic Aptituda Tast and alghth-arada 
National Assasamt nt ot Educational Progroas math aeorti . Many sfatas with conslstantly high tast 
storot spend wall below average par popll on adocatlon. tn contrast, the states that spend the 
nmt have among the lowest test scores. . 
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'RANK 


New Jersey 


$9,159 


39 


14 


, New York 


8.500 


42 


22 


District of Columbia 


8.210 


49 


42 


Connecticut 


7.914 


35 


11 


1 Alaska 


7,877 


31 


NA 


. Rhode Island 


6.989 


43 


23 


i Pennsylvania 


6.534 


45 


14 


: Massachusetts 


6.351 


33 


12 


; Maryland 


6.114 


32 


25 


I Deiaware 


6.016 


37 


27 


! Wisconsin 


5.946 


7 


6 


?: Maine 
4 Vermont 


5.894 


41 


4 


5.740 


36 


NA 


^Ohio 

' New Hampstiire 


5.639 


24 


18 


5.504 


28 


4 


f Virginia 


5.360 


38 


18 


! Oregon 


5.291 


26 


NA 


Minnesota 


5.260 


3 


3 


' Michigan 


5.237 


20 


18 


■ Wyomino 


5.255 


21 
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Montana 


5.184 


19 


NA 


Florida 


5.154 


40 


31 


^ Illinois 


5.062 


10 


NA 


. Indiana 


5.051 


47 


17 


- West Virginia 


5.046 


27 


34 


Washington 


5.045 


30 


NA 
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5.009 
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44 
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4.860 
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4,839 
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California 


4.826 


34 




Colorado 


4.809 


23 
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North Carolina 


4.802 


48 


34 


Nevada 


4.564 


29 


NA 


New Mexico 


4.446 


15 


31 


Missouri 


4.415 


13 


16 


Kentucky 


4.390 


18 


28 


NetKaska 


4.381 
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South Carolina 


4.327 


51 


29 


Texas 


4;^ 


46 


25 


Arizona 


4.231 


25 


23 


Louisiana 


4.012 


16 


40 


Oklahoma 


3.742 


11 


18 


South Dakota 


3.730 
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NA 


Tennessee 


3.707 
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34 


North Dakota 


3.685 
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Alabama 


3.648 


14 


39 


Arkansas 


3.334 


17 


38 


Mississippi 


3.322 


12 


41 


Idaho 


3.200 


22 
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2.993 
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Source: The Wall Street Journal 
Page A14, June 22. 1993 
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Figure Captions 

Figure 1. Plotting average within state Per Pupil Expenditures on education against the 
rank of each state based on its average SAT score seems to support the Heritage 
Foundation's inference that "Money Doesn't Help." 

Figure 2. When state NAEP ranks are compared with average within state Per Pupil 
Expenditures on education a small positive effect is clear. 
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A state's SAT rank worsens by six places for 
every thousand dollars spent on students 
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Per Pupil Expenditures 
(in $1,000) 
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Figure 1 



Worst 
ranking 
state 



A state's NAEP ranking improves two places for 
every thousand dollars spent on students 
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Figure 2 
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